{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a simple agent with TextWorld\n",
    "This tutorial outlines the steps to build an agent that learns how to play __choice-based__ text-based games generated with TextWorld."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prerequisite\n",
    "To run this notebook you need [PyTorch](https://pytorch.org/) (tested with v0.4.1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning challenges\n",
    "Training an agent such that it can learn how to play text-based games is not trivial. Among other challenges, we have to deal with\n",
    "\n",
    "1. a combinatorial action space (that grows w.r.t. vocabulary)\n",
    "2. a really sparse reward signal.\n",
    "\n",
    "To ease the learning process, we will be requesting additional information alongside the game's narrative (as covered in [Playing TextWorld generated games with OpenAI Gym](Playing%20TextWorld%20generated%20games%20with%20OpenAI%20Gym.ipynb#Interact-with-the-game)). More specifically, we will request the following information:\n",
    "\n",
    "- __Description__:\n",
    "For every game state, we will get the output of the `look` command which describes the current location;\n",
    "\n",
    "- __Inventory__:\n",
    "For every game state, we will get the output of the `inventory` command which describes the player's inventory;\n",
    "\n",
    "- __Admissible commands__:\n",
    "For every game state, we will get the list of commands guaranteed to be understood by the game interpreter;\n",
    "\n",
    "- __Intermediate reward__:\n",
    "For every game state, we will get an intermediate reward which can either be:\n",
    "  - __-1__: last action needs to be undone before resuming the quest\n",
    "  -  __0__: last action didn't affect the quest\n",
    "  -  __1__: last action brought us closer to completing the quest\n",
    "\n",
    "- __Entities__:\n",
    "For every game, we will get a list of entity names that the agent can interact with.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple test games\n",
    "We can use TextWorld to generate a few simple games with the following handcrafted world\n",
    "```\n",
    "                     Bathroom\n",
    "                        +\n",
    "                        |\n",
    "                        +\n",
    "    Bedroom +-(d1)-+ Kitchen +--(d2)--+ Backyard\n",
    "      (P)               +                  +\n",
    "                        |                  |\n",
    "                        +                  +\n",
    "                   Living Room           Garden\n",
    "```\n",
    "where the goal is always to retrieve a hidden food item and put it on the stove which located in the kitchen. One can lose the game if it eats the food item instead of putting it on the stove!\n",
    "\n",
    "Using `tw-make tw-simple ...` (see `make_games.sh` for the exact commands), we generated the following 7 games:\n",
    "\n",
    "| gamefile | description |\n",
    "| -------- | ----------- |\n",
    "| `games/rewardsDense_goalDetailed.ulx` | dense reward + detailed instructions |\n",
    "| `games/rewardsBalanced_goalDetailed.ulx` | balanced rewards + detailed instructions |\n",
    "| `games/rewardsSparse_goalDetailed.ulx` | sparse rewards + detailed instructions |\n",
    "| `games/rewardsDense_goalBrief.ulx` | dense rewards + no instructions but the goal is mentionned |\n",
    "| `games/rewardsBalanced_goalBrief.ulx` | balanced rewards + no instructions but the goal is mentionned |\n",
    "| `games/rewardsSparse_goalBrief.ulx` | sparse rewards + no instructions but the goal is mentionned |\n",
    "| `games/rewardsSparse_goalNone.ulx` | sparse rewards + no instructions/goal<br>_Hint: there's an hidden note in the game that describes the goal!_ |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the random baseline\n",
    "Let's start with building an agent that simply selects an admissible command at random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Mapping, Any\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import textworld.gym\n",
    "\n",
    "\n",
    "class RandomAgent(textworld.gym.Agent):\n",
    "    \"\"\" Agent that randomly selects a command from the admissible ones. \"\"\"\n",
    "    def __init__(self, seed=1234):\n",
    "        self.seed = seed\n",
    "        self.rng = np.random.RandomState(self.seed)\n",
    "\n",
    "    @property\n",
    "    def infos_to_request(self) -> textworld.EnvInfos:\n",
    "        return textworld.EnvInfos(admissible_commands=True)\n",
    "    \n",
    "    def act(self, obs: str, score: int, done: bool, infos: Mapping[str, Any]) -> str:\n",
    "        return self.rng.choice(infos[\"admissible_commands\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Play function\n",
    "Let's write a simple function to play a text-based game using an agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from glob import glob\n",
    "\n",
    "import gym\n",
    "import textworld.gym\n",
    "\n",
    "\n",
    "def play(agent, path, max_step=100, nb_episodes=10, verbose=True):\n",
    "    infos_to_request = agent.infos_to_request\n",
    "    infos_to_request.max_score = True  # Needed to normalize the scores.\n",
    "    \n",
    "    gamefiles = [path]\n",
    "    if os.path.isdir(path):\n",
    "        gamefiles = glob(os.path.join(path, \"*.ulx\"))\n",
    "        \n",
    "    env_id = textworld.gym.register_games(gamefiles,\n",
    "                                          request_infos=infos_to_request,\n",
    "                                          max_episode_steps=max_step)\n",
    "    env = gym.make(env_id)  # Create a Gym environment to play the text game.\n",
    "    if verbose:\n",
    "        if os.path.isdir(path):\n",
    "            print(os.path.dirname(path), end=\"\")\n",
    "        else:\n",
    "            print(os.path.basename(path), end=\"\")\n",
    "        \n",
    "    # Collect some statistics: nb_steps, final reward.\n",
    "    avg_moves, avg_scores, avg_norm_scores = [], [], []\n",
    "    for no_episode in range(nb_episodes):\n",
    "        obs, infos = env.reset()  # Start new episode.\n",
    "\n",
    "        score = 0\n",
    "        done = False\n",
    "        nb_moves = 0\n",
    "        while not done:\n",
    "            command = agent.act(obs, score, done, infos)\n",
    "            obs, score, done, infos = env.step(command)\n",
    "            nb_moves += 1\n",
    "        \n",
    "        agent.act(obs, score, done, infos)  # Let the agent know the game is done.\n",
    "                \n",
    "        if verbose:\n",
    "            print(\".\", end=\"\")\n",
    "        avg_moves.append(nb_moves)\n",
    "        avg_scores.append(score)\n",
    "        avg_norm_scores.append(score / infos[\"max_score\"])\n",
    "\n",
    "    env.close()\n",
    "    msg = \"  \\tavg. steps: {:5.1f}; avg. score: {:4.1f} / {}.\"\n",
    "    if verbose:\n",
    "        if os.path.isdir(path):\n",
    "            print(msg.format(np.mean(avg_moves), np.mean(avg_norm_scores), 1))\n",
    "        else:\n",
    "            print(msg.format(np.mean(avg_moves), np.mean(avg_scores), infos[\"max_score\"]))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the random agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rewardsDense_goalDetailed.ulx..........  \tavg. steps: 100.0; avg. score:  4.2 / 11.\n",
      "rewardsBalanced_goalDetailed.ulx..........  \tavg. steps: 100.0; avg. score:  0.7 / 5.\n",
      "rewardsSparse_goalDetailed.ulx..........  \tavg. steps: 100.0; avg. score:  0.0 / 1.\n"
     ]
    }
   ],
   "source": [
    "# We report the score and steps averaged over 10 playthroughs.\n",
    "play(RandomAgent(), \"./games/rewardsDense_goalDetailed.ulx\")    # Dense rewards\n",
    "play(RandomAgent(), \"./games/rewardsBalanced_goalDetailed.ulx\") # Balanced rewards\n",
    "play(RandomAgent(), \"./games/rewardsSparse_goalDetailed.ulx\")   # Sparse rewards"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural agent\n",
    "\n",
    "Now, let's create an agent that can learn to play text-based games. The agent will be trained to select a command from the list of admissible commands given the current game's narrative, inventory, and room description.\n",
    "\n",
    "Here's the implementation of that learning agent that uses [PyTorch](https://pytorch.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from typing import List, Mapping, Any, Optional\n",
    "from collections import defaultdict\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import textworld\n",
    "import textworld.gym\n",
    "from textworld import EnvInfos\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "\n",
    "class CommandScorer(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size):\n",
    "        super(CommandScorer, self).__init__()\n",
    "        torch.manual_seed(42)  # For reproducibility\n",
    "        self.embedding    = nn.Embedding(input_size, hidden_size)\n",
    "        self.encoder_gru  = nn.GRU(hidden_size, hidden_size)\n",
    "        self.cmd_encoder_gru  = nn.GRU(hidden_size, hidden_size)\n",
    "        self.state_gru    = nn.GRU(hidden_size, hidden_size)\n",
    "        self.hidden_size  = hidden_size\n",
    "        self.state_hidden = torch.zeros(1, 1, hidden_size, device=device)\n",
    "        self.critic       = nn.Linear(hidden_size, 1)\n",
    "        self.att_cmd      = nn.Linear(hidden_size * 2, 1)\n",
    "\n",
    "    def forward(self, obs, commands, **kwargs):\n",
    "        input_length = obs.size(0)\n",
    "        batch_size = obs.size(1)\n",
    "        nb_cmds = commands.size(1)\n",
    "\n",
    "        embedded = self.embedding(obs)\n",
    "        encoder_output, encoder_hidden = self.encoder_gru(embedded)\n",
    "        state_output, state_hidden = self.state_gru(encoder_hidden, self.state_hidden)\n",
    "        self.state_hidden = state_hidden\n",
    "        value = self.critic(state_output)\n",
    "\n",
    "        # Attention network over the commands.\n",
    "        cmds_embedding = self.embedding.forward(commands)\n",
    "        _, cmds_encoding_last_states = self.cmd_encoder_gru.forward(cmds_embedding)  # 1 x cmds x hidden\n",
    "\n",
    "        # Same observed state for all commands.\n",
    "        cmd_selector_input = torch.stack([state_hidden] * nb_cmds, 2)  # 1 x batch x cmds x hidden\n",
    "\n",
    "        # Same command choices for the whole batch.\n",
    "        cmds_encoding_last_states = torch.stack([cmds_encoding_last_states] * batch_size, 1)  # 1 x batch x cmds x hidden\n",
    "\n",
    "        # Concatenate the observed state and command encodings.\n",
    "        cmd_selector_input = torch.cat([cmd_selector_input, cmds_encoding_last_states], dim=-1)\n",
    "\n",
    "        # Compute one score per command.\n",
    "        scores = F.relu(self.att_cmd(cmd_selector_input)).squeeze(-1)  # 1 x Batch x cmds\n",
    "\n",
    "        probs = F.softmax(scores, dim=2)  # 1 x Batch x cmds\n",
    "        index = probs[0].multinomial(num_samples=1).unsqueeze(0) # 1 x batch x indx\n",
    "        return scores, index, value\n",
    "\n",
    "    def reset_hidden(self, batch_size):\n",
    "        self.state_hidden = torch.zeros(1, batch_size, self.hidden_size, device=device)\n",
    "\n",
    "\n",
    "class NeuralAgent:\n",
    "    \"\"\" Simple Neural Agent for playing TextWorld games. \"\"\"\n",
    "    MAX_VOCAB_SIZE = 1000\n",
    "    UPDATE_FREQUENCY = 10\n",
    "    LOG_FREQUENCY = 1000\n",
    "    GAMMA = 0.9\n",
    "    \n",
    "    def __init__(self) -> None:\n",
    "        self._initialized = False\n",
    "        self._epsiode_has_started = False\n",
    "        self.id2word = [\"<PAD>\", \"<UNK>\"]\n",
    "        self.word2id = {w: i for i, w in enumerate(self.id2word)}\n",
    "        \n",
    "        self.model = CommandScorer(input_size=self.MAX_VOCAB_SIZE, hidden_size=128)\n",
    "        self.optimizer = optim.Adam(self.model.parameters(), 0.00003)\n",
    "        \n",
    "        self.mode = \"test\"\n",
    "    \n",
    "    def train(self):\n",
    "        self.mode = \"train\"\n",
    "        self.stats = {\"max\": defaultdict(list), \"mean\": defaultdict(list)}\n",
    "        self.transitions = []\n",
    "        self.model.reset_hidden(1)\n",
    "        self.last_score = 0\n",
    "        self.no_train_step = 0\n",
    "    \n",
    "    def test(self):\n",
    "        self.mode = \"test\"\n",
    "        self.model.reset_hidden(1)\n",
    "        \n",
    "    @property\n",
    "    def infos_to_request(self) -> EnvInfos:\n",
    "        return EnvInfos(description=True, inventory=True, admissible_commands=True,\n",
    "                        won=True, lost=True)\n",
    "    \n",
    "    def _get_word_id(self, word):\n",
    "        if word not in self.word2id:\n",
    "            if len(self.word2id) >= self.MAX_VOCAB_SIZE:\n",
    "                return self.word2id[\"<UNK>\"]\n",
    "            \n",
    "            self.id2word.append(word)\n",
    "            self.word2id[word] = len(self.word2id)\n",
    "            \n",
    "        return self.word2id[word]\n",
    "            \n",
    "    def _tokenize(self, text):\n",
    "        # Simple tokenizer: strip out all non-alphabetic characters.\n",
    "        text = re.sub(\"[^a-zA-Z0-9\\- ]\", \" \", text)\n",
    "        word_ids = list(map(self._get_word_id, text.split()))\n",
    "        return word_ids\n",
    "\n",
    "    def _process(self, texts):\n",
    "        texts = list(map(self._tokenize, texts))\n",
    "        max_len = max(len(l) for l in texts)\n",
    "        padded = np.ones((len(texts), max_len)) * self.word2id[\"<PAD>\"]\n",
    "\n",
    "        for i, text in enumerate(texts):\n",
    "            padded[i, :len(text)] = text\n",
    "\n",
    "        padded_tensor = torch.from_numpy(padded).type(torch.long).to(device)\n",
    "        padded_tensor = padded_tensor.permute(1, 0) # Batch x Seq => Seq x Batch\n",
    "        return padded_tensor\n",
    "      \n",
    "    def _discount_rewards(self, last_values):\n",
    "        returns, advantages = [], []\n",
    "        R = last_values.data\n",
    "        for t in reversed(range(len(self.transitions))):\n",
    "            rewards, _, _, values = self.transitions[t]\n",
    "            R = rewards + self.GAMMA * R\n",
    "            adv = R - values\n",
    "            returns.append(R)\n",
    "            advantages.append(adv)\n",
    "            \n",
    "        return returns[::-1], advantages[::-1]\n",
    "\n",
    "    def act(self, obs: str, score: int, done: bool, infos: Mapping[str, Any]) -> Optional[str]:\n",
    "        \n",
    "        # Build agent's observation: feedback + look + inventory.\n",
    "        input_ = \"{}\\n{}\\n{}\".format(obs, infos[\"description\"], infos[\"inventory\"])\n",
    "        \n",
    "        # Tokenize and pad the input and the commands to chose from.\n",
    "        input_tensor = self._process([input_])\n",
    "        commands_tensor = self._process(infos[\"admissible_commands\"])\n",
    "        \n",
    "        # Get our next action and value prediction.\n",
    "        outputs, indexes, values = self.model(input_tensor, commands_tensor)\n",
    "        action = infos[\"admissible_commands\"][indexes[0]]\n",
    "        \n",
    "        if self.mode == \"test\":\n",
    "            if done:\n",
    "                self.model.reset_hidden(1)\n",
    "            return action\n",
    "        \n",
    "        self.no_train_step += 1\n",
    "        \n",
    "        if self.transitions:\n",
    "            reward = score - self.last_score  # Reward is the gain/loss in score.\n",
    "            self.last_score = score\n",
    "            if infos[\"won\"]:\n",
    "                reward += 100\n",
    "            if infos[\"lost\"]:\n",
    "                reward -= 100\n",
    "                \n",
    "            self.transitions[-1][0] = reward  # Update reward information.\n",
    "        \n",
    "        self.stats[\"max\"][\"score\"].append(score)\n",
    "        if self.no_train_step % self.UPDATE_FREQUENCY == 0:\n",
    "            # Update model\n",
    "            returns, advantages = self._discount_rewards(values)\n",
    "            \n",
    "            loss = 0\n",
    "            for transition, ret, advantage in zip(self.transitions, returns, advantages):\n",
    "                reward, indexes_, outputs_, values_ = transition\n",
    "                \n",
    "                advantage        = advantage.detach() # Block gradients flow here.\n",
    "                probs            = F.softmax(outputs_, dim=2)\n",
    "                log_probs        = torch.log(probs)\n",
    "                log_action_probs = log_probs.gather(2, indexes_)\n",
    "                policy_loss      = (-log_action_probs * advantage).sum()\n",
    "                value_loss       = (.5 * (values_ - ret) ** 2.).sum()\n",
    "                entropy     = (-probs * log_probs).sum()\n",
    "                loss += policy_loss + 0.5 * value_loss - 0.1 * entropy\n",
    "                \n",
    "                self.stats[\"mean\"][\"reward\"].append(reward)\n",
    "                self.stats[\"mean\"][\"policy\"].append(policy_loss.item())\n",
    "                self.stats[\"mean\"][\"value\"].append(value_loss.item())\n",
    "                self.stats[\"mean\"][\"entropy\"].append(entropy.item())\n",
    "                self.stats[\"mean\"][\"confidence\"].append(torch.exp(log_action_probs).item())\n",
    "            \n",
    "            if self.no_train_step % self.LOG_FREQUENCY == 0:\n",
    "                msg = \"{}. \".format(self.no_train_step)\n",
    "                msg += \"  \".join(\"{}: {:.3f}\".format(k, np.mean(v)) for k, v in self.stats[\"mean\"].items())\n",
    "                msg += \"  \" + \"  \".join(\"{}: {}\".format(k, np.max(v)) for k, v in self.stats[\"max\"].items())\n",
    "                msg += \"  vocab: {}\".format(len(self.id2word))\n",
    "                print(msg)\n",
    "                self.stats = {\"max\": defaultdict(list), \"mean\": defaultdict(list)}\n",
    "            \n",
    "            loss.backward()\n",
    "            nn.utils.clip_grad_norm_(self.model.parameters(), 40)\n",
    "            self.optimizer.step()\n",
    "            self.optimizer.zero_grad()\n",
    "        \n",
    "            self.transitions = []\n",
    "            self.model.reset_hidden(1)\n",
    "        else:\n",
    "            # Keep information about transitions for Truncated Backpropagation Through Time.\n",
    "            self.transitions.append([None, indexes, outputs, values])  # Reward will be set on the next call\n",
    "        \n",
    "        if done:\n",
    "            self.last_score = 0  # Will be starting a new episode. Reset the last score.\n",
    "        \n",
    "        return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training the neural agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rewardsDense_goalDetailed.ulx..........  \tavg. steps:  95.4; avg. score:  4.5 / 11.\n"
     ]
    }
   ],
   "source": [
    "agent = NeuralAgent()\n",
    "play(agent, \"./games/rewardsDense_goalDetailed.ulx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unsurprisingly, the result is not much different from what the random agent can get.\n",
    "\n",
    "Let's train the agent for a few episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training\n",
      "1000. reward: -0.054  policy: 0.134  value: 5.593  entropy: 2.381  confidence: 0.096  score: 10  vocab: 327\n",
      "2000. reward: 0.037  policy: 0.027  value: 0.056  entropy: 2.367  confidence: 0.096  score: 6  vocab: 331\n",
      "3000. reward: 0.050  policy: 0.125  value: 0.095  entropy: 2.396  confidence: 0.094  score: 10  vocab: 331\n",
      "4000. reward: 0.042  policy: 0.059  value: 0.074  entropy: 2.357  confidence: 0.099  score: 7  vocab: 331\n",
      "5000. reward: -0.046  policy: -0.118  value: 5.368  entropy: 2.476  confidence: 0.088  score: 10  vocab: 331\n",
      "6000. reward: 0.056  policy: 0.021  value: 0.118  entropy: 2.447  confidence: 0.092  score: 7  vocab: 331\n",
      "7000. reward: 0.060  policy: 0.056  value: 0.110  entropy: 2.466  confidence: 0.091  score: 10  vocab: 331\n",
      "8000. reward: 0.062  policy: 0.036  value: 0.097  entropy: 2.467  confidence: 0.094  score: 8  vocab: 331\n",
      "9000. reward: 0.053  policy: 0.019  value: 0.082  entropy: 2.413  confidence: 0.098  score: 8  vocab: 331\n",
      "10000. reward: 0.048  policy: -0.027  value: 0.090  entropy: 2.439  confidence: 0.095  score: 6  vocab: 331\n",
      "11000. reward: 0.057  policy: 0.046  value: 0.092  entropy: 2.361  confidence: 0.103  score: 10  vocab: 331\n",
      "12000. reward: 0.059  policy: 0.104  value: 0.098  entropy: 2.413  confidence: 0.097  score: 10  vocab: 331\n",
      "13000. reward: 0.058  policy: 0.025  value: 0.111  entropy: 2.452  confidence: 0.095  score: 10  vocab: 331\n",
      "14000. reward: 0.063  policy: -0.019  value: 0.105  entropy: 2.349  confidence: 0.103  score: 10  vocab: 331\n",
      "15000. reward: 0.058  policy: 0.014  value: 0.116  entropy: 2.420  confidence: 0.100  score: 10  vocab: 331\n",
      "16000. reward: 0.060  policy: -0.047  value: 0.085  entropy: 2.409  confidence: 0.099  score: 7  vocab: 331\n",
      "17000. reward: 0.053  policy: -0.060  value: 0.091  entropy: 2.386  confidence: 0.107  score: 7  vocab: 331\n",
      "18000. reward: -0.031  policy: -0.053  value: 5.335  entropy: 2.581  confidence: 0.087  score: 10  vocab: 332\n",
      "19000. reward: -0.049  policy: -1.100  value: 16.763  entropy: 2.415  confidence: 0.103  score: 10  vocab: 333\n",
      "20000. reward: -0.033  policy: -0.166  value: 5.583  entropy: 2.475  confidence: 0.100  score: 10  vocab: 334\n",
      "21000. reward: 0.066  policy: 0.027  value: 0.104  entropy: 2.490  confidence: 0.099  score: 8  vocab: 334\n",
      "22000. reward: 0.073  policy: 0.016  value: 0.119  entropy: 2.451  confidence: 0.099  score: 10  vocab: 334\n",
      "23000. reward: -0.249  policy: -4.069  value: 69.326  entropy: 2.420  confidence: 0.103  score: 10  vocab: 337\n",
      "24000. reward: -0.123  policy: -1.768  value: 33.473  entropy: 2.552  confidence: 0.099  score: 10  vocab: 339\n",
      "25000. reward: -0.134  policy: -2.818  value: 44.121  entropy: 2.446  confidence: 0.114  score: 10  vocab: 341\n",
      "26000. reward: -0.111  policy: -0.973  value: 22.873  entropy: 2.438  confidence: 0.116  score: 10  vocab: 342\n",
      "27000. reward: -0.546  policy: -7.568  value: 121.705  entropy: 2.337  confidence: 0.127  score: 10  vocab: 347\n",
      "28000. reward: -0.438  policy: -3.946  value: 72.391  entropy: 2.262  confidence: 0.142  score: 10  vocab: 350\n",
      "29000. reward: 0.002  policy: -0.253  value: 9.844  entropy: 2.288  confidence: 0.144  score: 10  vocab: 350\n",
      "30000. reward: -0.416  policy: -4.680  value: 79.016  entropy: 2.268  confidence: 0.155  score: 10  vocab: 352\n",
      "31000. reward: -0.202  policy: -2.496  value: 39.720  entropy: 2.207  confidence: 0.165  score: 10  vocab: 356\n",
      "32000. reward: -0.620  policy: -6.433  value: 123.878  entropy: 2.173  confidence: 0.180  score: 10  vocab: 365\n",
      "33000. reward: -0.316  policy: -3.617  value: 62.929  entropy: 2.229  confidence: 0.169  score: 10  vocab: 368\n",
      "34000. reward: -0.311  policy: -3.923  value: 65.803  entropy: 2.158  confidence: 0.175  score: 10  vocab: 371\n",
      "35000. reward: -0.418  policy: -4.113  value: 73.234  entropy: 2.198  confidence: 0.184  score: 10  vocab: 373\n",
      "36000. reward: -1.062  policy: -12.780  value: 220.467  entropy: 2.197  confidence: 0.188  score: 10  vocab: 376\n",
      "37000. reward: -1.038  policy: -10.782  value: 177.655  entropy: 2.138  confidence: 0.196  score: 10  vocab: 379\n",
      "38000. reward: -0.933  policy: -7.264  value: 128.102  entropy: 2.116  confidence: 0.196  score: 10  vocab: 382\n",
      "39000. reward: -0.833  policy: -7.505  value: 141.487  entropy: 2.108  confidence: 0.194  score: 10  vocab: 386\n",
      "40000. reward: -1.044  policy: -11.601  value: 198.037  entropy: 2.094  confidence: 0.200  score: 10  vocab: 388\n",
      "41000. reward: -0.313  policy: -4.148  value: 68.481  entropy: 2.146  confidence: 0.193  score: 10  vocab: 389\n",
      "42000. reward: -0.622  policy: -8.270  value: 127.728  entropy: 2.131  confidence: 0.192  score: 10  vocab: 393\n",
      "43000. reward: -0.822  policy: -10.149  value: 175.033  entropy: 2.111  confidence: 0.204  score: 10  vocab: 397\n",
      "Trained in 2130.74 secs\n"
     ]
    }
   ],
   "source": [
    "from time import time\n",
    "agent = NeuralAgent()\n",
    "\n",
    "print(\"Training\")\n",
    "agent.train()  # Tell the agent it should update its parameters.\n",
    "starttime = time()\n",
    "play(agent, \"./games/rewardsDense_goalDetailed.ulx\", nb_episodes=500, verbose=False)  # Dense rewards game.\n",
    "print(\"Trained in {:.2f} secs\".format(time() - starttime))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the trained agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rewardsDense_goalDetailed.ulx..........  \tavg. steps:  74.4; avg. score: 10.0 / 11.\n"
     ]
    }
   ],
   "source": [
    "# We report the score and steps averaged over 10 playthroughs.\n",
    "agent.test()\n",
    "play(agent, \"./games/rewardsDense_goalDetailed.ulx\")  # Dense rewards game."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, since we trained on that single simple game, it's not surprinsing the agent can achieve a high score on it. It would be more interesting to evaluate the generalization capability of the agent.\n",
    "\n",
    "To do so, we are going to test the agent on another game drawn from the same game distribution (i.e. same world but the goal is to pick another food item). Let's generate `games/another_game.ulx` with the same rewards density (`--rewards dense`) and the same goal description (`--goal detailed`), but using `--seed 1` and without the `--test` flag (to make sure the game is not part of the test set since `games/rewardsDense_goalDetailed.ulx` is)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed: 1\n",
      "Game generated: /data/src/TextWorld/notebooks/games/another_game.ulx\n",
      "\n",
      "Objective:\n",
      "Hey, thanks for coming over to the TextWorld today, there is something I need you to do for me. First, it would be a great idea if you could doublecheck that the antique trunk inside the bedroom is wide open. And then, recover the old key from the antique trunk in the bedroom. If you can get your hands on the old key, unlock the wooden door within the bedroom. And then, open the wooden door in the bedroom. Having pulled open the wooden door, venture east. With that accomplished, go to the south. And then, take the milk from the couch within the living room. After that, make an attempt to travel north. Then, sit the milk on the stove. Alright, thanks!\n",
      "\n",
      "Walkthrough:\n",
      "open antique trunk > take old key from antique trunk > unlock wooden door with old key > open wooden door > go east > go south > take milk from couch > go north > put milk on stove\n",
      "\n",
      "-= Stats =-\n",
      "Nb. locations: 6\n",
      "Nb. objects: 28\n"
     ]
    }
   ],
   "source": [
    "!tw-make tw-simple --rewards dense --goal detailed --seed 1 --output games/another_game.ulx -v -f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "another_game.ulx..........  \tavg. steps: 100.0; avg. score:  4.1 / 9.\n",
      "another_game.ulx..........  \tavg. steps: 100.0; avg. score:  5.3 / 9.\n"
     ]
    }
   ],
   "source": [
    "# We report the score and steps averaged over 10 playthroughs.\n",
    "play(RandomAgent(), \"./games/another_game.ulx\")\n",
    "play(agent, \"./games/another_game.ulx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see the trained agent barely does better than the random agent. In order to improve the agent's generalization capability, we should train it on many different games drawn from the game distribution.\n",
    "\n",
    "One could use the following command to easily generate 100 training games:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed: 3\n",
      "Global seed: 1\n",
      "Global seed: 2\n",
      "Global seed: 4\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-ekDZtbGXIbO5FKp8.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-D8gMTlO8cPoEtgZx.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-E5eLHkaXFk6BSgR1.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-ek06H8B7uqoYFVEy.ulx\n",
      "Global seed: 5\n",
      "Global seed: 6\n",
      "Global seed: 7\n",
      "Global seed: 8\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-7KpYUDDdckE0cBqZ.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-o2RVTmrEi6R5T3p0.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-68kvf8x7TBd9Iq0P.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-KJODI168SvJVFM9x.ulx\n",
      "Global seed: 9\n",
      "Global seed: 10\n",
      "Global seed: 11\n",
      "Global seed: 12\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-NPQ8TkJ9i2x6fYDM.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-redEHVr6CmKYhrJg.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-Q9nDu630U5j3tqBG.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-Qbq3h3VWFkPdtogB.ulx\n",
      "Global seed: 13\n",
      "Global seed: 14\n",
      "Global seed: 15\n",
      "Global seed: 16\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-jROVIEqEIya6Tr0L.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-1QKVfg6YhRb1ul1e.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-7yGrcV9pTE8DF75n.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-6GMVtjVYF5QRupyN.ulx\n",
      "Global seed: 17\n",
      "Global seed: 18\n",
      "Global seed: 19\n",
      "Global seed: 20\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-nrEoHEqgUba7U1nx.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-Mn8oTkr2fvv8TX1.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-VLpEiW2msKJpTZ7J.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-o8WVtobyuRR0Uqv5.ulx\n",
      "Global seed: 21\n",
      "Global seed: 22\n",
      "Global seed: 23\n",
      "Global seed: 24\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-JGJdTBvVHrpBiVy5.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-M7MmCGG5i6kES1kV.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-6GZ3CqJvCX9rfev3.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-1MB8fmosEL9HNEv.ulx\n",
      "Global seed: 25\n",
      "Global seed: 26\n",
      "Global seed: 27\n",
      "Global seed: 28\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-mn3JuMrnfPNNI5K5.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-P6RdC912uMrbcXBX.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-pGedtxVJsYDxsGaV.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-R2bxU9dJUK0LCxk0.ulx\n",
      "Global seed: 29\n",
      "Global seed: 30\n",
      "Global seed: 31\n",
      "Global seed: 32\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-W1qJibX5FqLRS3kL.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-eDi7Z2iEJ7FdL9.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-6WNBiZyofr8mu6MG.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-QlDlS6DxCMnVu81D.ulx\n",
      "Global seed: 33\n",
      "Global seed: 34\n",
      "Global seed: 35\n",
      "Global seed: 36\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-5rjrFkZEiyOIj1y.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-dZ2oU2KnflPksnEY.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-KrNpTrdMtdqVUKOB.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-J9ylcQmmc190Cgn3.ulx\n",
      "Global seed: 37\n",
      "Global seed: 38\n",
      "Global seed: 39\n",
      "Global seed: 40\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-M8xnUOBkulX7HNQX.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-535aCDxgu5MqF7R1.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-nOVQCNlrINVxIDje.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-mDjNimx7S5a5tmZ7.ulx\n",
      "Global seed: 41\n",
      "Global seed: 42\n",
      "Global seed: 44\n",
      "Global seed: 43\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-566vUvgjfXgU9GK.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-67XKC3EyH2riOk8.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-mbJLuQBoCbbkUv1n.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-El9oUXJWIYVgF1jy.ulx\n",
      "Global seed: 45\n",
      "Global seed: 46\n",
      "Global seed: 47\n",
      "Global seed: 48\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-M2vecROLFVBxHBvl.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-mNB2tM2ohGmRIB2J.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-JBP7TQZ6fV7biZ9q.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-aernSYY1f7rrflvB.ulx\n",
      "Global seed: 49\n",
      "Global seed: 50\n",
      "Global seed: 51\n",
      "Global seed: 52\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-5VVVSK6fGjlspnj.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-Oq9ns8K0uK5EhJvr.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-Mj8NTxYWso7LfmN9.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-xqa9SlmxsDmVs8yb.ulx\n",
      "Global seed: 53\n",
      "Global seed: 54\n",
      "Global seed: 55\n",
      "Global seed: 56\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-GN1YHrQ6s6vBTnNb.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-EloyS7BZs8LRHNVn.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-XR5RUnmMIgOnCYyy.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-p1RLFX7MU7KjFy0d.ulx\n",
      "Global seed: 57\n",
      "Global seed: 58\n",
      "Global seed: 59\n",
      "Global seed: 60\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-WxnkhG07upKRhbNV.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-Kx32iybbtDGRFQl6.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-DWvMiBQvCR5sNkx.ulx\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-R7y2UJkBUeEqS8Bk.ulx\n",
      "Global seed: 61\n",
      "Global seed: 62\n",
      "Global seed: 63\n",
      "Global seed: 64\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-YnvLs6vYIWvWC51e.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-1EQPheDOiVB8I7m5.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-6VXXCVGKiEWkhPer.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-G6a7HeZPcJNGhLQ2.ulx\n",
      "Global seed: 65\n",
      "Global seed: 66\n",
      "Global seed: 67\n",
      "Global seed: 68\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-1jr7FeK9TgZCQ9n.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-0a72tMGVtQnPSvMR.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-2OOjSlaNUBKcgLy.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-1vbrCekqT6xMcMdP.ulx\n",
      "Global seed: 69\n",
      "Global seed: 70\n",
      "Global seed: 71\n",
      "Global seed: 72\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-6kRniM7gfXQNCyj6.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-kGDLudo8fXeJH1Yg.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-RvQYhbdKfMyqS5Wp.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-aMq9U18aFMlOHVEe.ulx\n",
      "Global seed: 74\n",
      "Global seed: 73\n",
      "Global seed: 75\n",
      "Global seed: 76\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-8koWh6KqcgNPI8Ox.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-165ai83atZWatMRa.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-Eykru3m8ilOGiBnx.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-vKm0IdN8c0VRfJl1.ulx\n",
      "Global seed: 77\n",
      "Global seed: 78\n",
      "Global seed: 79\n",
      "Global seed: 80\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-D3PEFjpNIQykuqra.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-gPyQi5vkhdQOUV0J.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-jYPJukgls6oZf6qG.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-eRNBC9q7tr5Vi3Mv.ulx\n",
      "Global seed: 81\n",
      "Global seed: 82\n",
      "Global seed: 83\n",
      "Global seed: 84\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-0lvotxEKtYZ6umyJ.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-3bj9creDs3V0IKKN.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-qZDfWDNTJmQiLvq.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-6yPJsEJMTeR6fpPR.ulx\n",
      "Global seed: 86\n",
      "Global seed: 85\n",
      "Global seed: 87\n",
      "Global seed: 88\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-WrmvTdgGc05rh3eZ.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-g0GEcMQ5CWaeF3nJ.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-5PBeh9J3IX5XUjL3.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-DvPBs6OphW6nCnQ3.ulx\n",
      "Global seed: 89\n",
      "Global seed: 90\n",
      "Global seed: 91\n",
      "Global seed: 92\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-eBxGu3KeiYjbHDl9.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-XElPhYRjT8gWs1eM.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-3qLvIOZ9c66Igby.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-OWRZsvM8h6XyIpQy.ulx\n",
      "Global seed: 93\n",
      "Global seed: 94\n",
      "Global seed: 95\n",
      "Global seed: 96\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-gkdXTMLPtKpeSY5J.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-n3OYtnV7IY8rTkBq.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-q3rdiVK5I1lrsNpj.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-WvgBsoOmhDObfRgL.ulx\n",
      "Global seed: 97\n",
      "Global seed: 98\n",
      "Global seed: 99\n",
      "Global seed: 100\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-xQNoFqlDs10nfr79.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-1dgPTd1ki5kQUkrn.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-o7PNc5j0URa5HYLb.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/training_games/tw-simple-rDense+gDetailed+train-house-GP-2KK5iXD8ueb3ikl6.ulx\n"
     ]
    }
   ],
   "source": [
    "! seq 1 100 | xargs -n1 -P4 tw-make tw-simple --rewards dense --goal detailed --output training_games/ --seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we train our agent on that set of training games."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training on 100 games\n",
      "1000. reward: 0.043  policy: 0.278  value: 0.076  entropy: 2.375  confidence: 0.097  score: 7  vocab: 456\n",
      "2000. reward: 0.044  policy: 0.118  value: 0.073  entropy: 2.372  confidence: 0.096  score: 7  vocab: 535\n",
      "3000. reward: -0.059  policy: -0.616  value: 13.598  entropy: 2.433  confidence: 0.092  score: 8  vocab: 604\n",
      "4000. reward: 0.036  policy: 0.022  value: 0.061  entropy: 2.313  confidence: 0.102  score: 5  vocab: 624\n",
      "5000. reward: 0.157  policy: 0.804  value: 10.740  entropy: 2.426  confidence: 0.094  score: 8  vocab: 637\n",
      "6000. reward: 0.044  policy: 0.046  value: 0.075  entropy: 2.394  confidence: 0.094  score: 6  vocab: 649\n",
      "7000. reward: 0.048  policy: 0.026  value: 0.081  entropy: 2.434  confidence: 0.092  score: 6  vocab: 673\n",
      "8000. reward: 0.058  policy: 0.052  value: 0.098  entropy: 2.386  confidence: 0.097  score: 10  vocab: 683\n",
      "9000. reward: 0.047  policy: -0.020  value: 0.075  entropy: 2.341  confidence: 0.101  score: 6  vocab: 684\n",
      "10000. reward: 0.052  policy: 0.084  value: 0.106  entropy: 2.387  confidence: 0.098  score: 8  vocab: 695\n",
      "11000. reward: 0.050  policy: -0.011  value: 0.086  entropy: 2.364  confidence: 0.101  score: 7  vocab: 696\n",
      "12000. reward: 0.057  policy: 0.115  value: 0.114  entropy: 2.404  confidence: 0.099  score: 8  vocab: 697\n",
      "13000. reward: 0.058  policy: -0.019  value: 0.098  entropy: 2.445  confidence: 0.095  score: 6  vocab: 702\n",
      "14000. reward: 0.056  policy: 0.014  value: 0.095  entropy: 2.436  confidence: 0.097  score: 6  vocab: 702\n",
      "15000. reward: -0.043  policy: -1.799  value: 23.996  entropy: 2.450  confidence: 0.091  score: 10  vocab: 704\n",
      "16000. reward: 0.174  policy: 1.156  value: 17.321  entropy: 2.390  confidence: 0.101  score: 10  vocab: 707\n",
      "17000. reward: 0.062  policy: 0.030  value: 0.090  entropy: 2.431  confidence: 0.098  score: 8  vocab: 709\n",
      "18000. reward: 0.186  policy: 2.070  value: 25.049  entropy: 2.518  confidence: 0.090  score: 8  vocab: 712\n",
      "19000. reward: -0.154  policy: -1.444  value: 24.538  entropy: 2.490  confidence: 0.093  score: 10  vocab: 716\n",
      "20000. reward: -0.048  policy: -0.391  value: 5.497  entropy: 2.477  confidence: 0.095  score: 8  vocab: 718\n",
      "21000. reward: 0.067  policy: 0.009  value: 0.133  entropy: 2.506  confidence: 0.095  score: 10  vocab: 718\n",
      "22000. reward: -0.251  policy: -2.577  value: 41.298  entropy: 2.489  confidence: 0.094  score: 10  vocab: 720\n",
      "23000. reward: 0.066  policy: -0.068  value: 0.115  entropy: 2.457  confidence: 0.100  score: 8  vocab: 721\n",
      "24000. reward: -0.042  policy: -1.519  value: 21.137  entropy: 2.529  confidence: 0.096  score: 8  vocab: 724\n",
      "25000. reward: 0.064  policy: 0.025  value: 0.126  entropy: 2.490  confidence: 0.104  score: 8  vocab: 725\n",
      "26000. reward: -0.032  policy: -1.725  value: 22.567  entropy: 2.542  confidence: 0.096  score: 10  vocab: 726\n",
      "27000. reward: 0.070  policy: 0.035  value: 0.150  entropy: 2.467  confidence: 0.102  score: 10  vocab: 728\n",
      "28000. reward: -0.126  policy: -0.930  value: 69.579  entropy: 2.438  confidence: 0.107  score: 10  vocab: 734\n",
      "29000. reward: -0.242  policy: -2.803  value: 47.557  entropy: 2.473  confidence: 0.107  score: 10  vocab: 735\n",
      "30000. reward: -0.229  policy: -1.673  value: 32.351  entropy: 2.428  confidence: 0.119  score: 10  vocab: 737\n",
      "31000. reward: -0.329  policy: -4.816  value: 79.937  entropy: 2.378  confidence: 0.127  score: 10  vocab: 740\n",
      "32000. reward: -0.006  policy: 0.579  value: 48.690  entropy: 2.364  confidence: 0.129  score: 10  vocab: 743\n",
      "33000. reward: 0.066  policy: -0.103  value: 0.163  entropy: 2.427  confidence: 0.120  score: 7  vocab: 748\n",
      "34000. reward: -0.117  policy: -3.919  value: 71.403  entropy: 2.477  confidence: 0.116  score: 10  vocab: 750\n",
      "35000. reward: -0.341  policy: -3.954  value: 59.400  entropy: 2.304  confidence: 0.139  score: 10  vocab: 753\n",
      "36000. reward: 0.066  policy: 0.007  value: 0.206  entropy: 2.345  confidence: 0.133  score: 10  vocab: 753\n",
      "37000. reward: 0.106  policy: -0.906  value: 77.322  entropy: 2.325  confidence: 0.135  score: 10  vocab: 756\n",
      "38000. reward: -0.344  policy: -2.403  value: 101.753  entropy: 2.384  confidence: 0.125  score: 10  vocab: 757\n",
      "39000. reward: -0.457  policy: -5.915  value: 87.720  entropy: 2.363  confidence: 0.129  score: 10  vocab: 761\n",
      "40000. reward: -0.020  policy: -0.941  value: 61.225  entropy: 2.384  confidence: 0.129  score: 10  vocab: 762\n",
      "41000. reward: -0.220  policy: -1.943  value: 36.295  entropy: 2.396  confidence: 0.132  score: 10  vocab: 765\n",
      "42000. reward: -0.550  policy: -7.035  value: 117.764  entropy: 2.247  confidence: 0.146  score: 10  vocab: 768\n",
      "43000. reward: -0.037  policy: -1.858  value: 25.157  entropy: 2.372  confidence: 0.126  score: 10  vocab: 768\n",
      "44000. reward: -0.230  policy: -1.500  value: 31.274  entropy: 2.397  confidence: 0.134  score: 10  vocab: 770\n",
      "45000. reward: -0.140  policy: -2.222  value: 32.777  entropy: 2.355  confidence: 0.131  score: 10  vocab: 770\n",
      "46000. reward: -0.320  policy: -2.607  value: 97.147  entropy: 2.434  confidence: 0.130  score: 10  vocab: 773\n",
      "47000. reward: -0.013  policy: -0.434  value: 63.372  entropy: 2.327  confidence: 0.143  score: 10  vocab: 775\n",
      "Trained in 2164.76 secs\n"
     ]
    }
   ],
   "source": [
    "from time import time\n",
    "agent = NeuralAgent()\n",
    "\n",
    "print(\"Training on 100 games\")\n",
    "agent.train()  # Tell the agent it should update its parameters.\n",
    "starttime = time()\n",
    "play(agent, \"./training_games/\", nb_episodes=100 * 5, verbose=False)  # Each game will be seen 5 times.\n",
    "print(\"Trained in {:.2f} secs\".format(time() - starttime))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating the agent on the test distribution\n",
    "We will generate 20 test games and evaluate the agent on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed: 3\n",
      "Global seed: 2\n",
      "Global seed: 4\n",
      "Global seed: 1\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-E5eLHkaXFk6BSgR1.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-D8gMTlO8cPoEtgZx.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-ek06H8B7uqoYFVEy.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-ekDZtbGXIbO5FKp8.ulx\n",
      "Global seed: 6\n",
      "Global seed: 5\n",
      "Global seed: 7\n",
      "Global seed: 8\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-o2RVTmrEi6R5T3p0.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-7KpYUDDdckE0cBqZ.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-68kvf8x7TBd9Iq0P.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-KJODI168SvJVFM9x.ulx\n",
      "Global seed: 10\n",
      "Global seed: 9\n",
      "Global seed: 11\n",
      "Global seed: 12\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-redEHVr6CmKYhrJg.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-NPQ8TkJ9i2x6fYDM.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-Q9nDu630U5j3tqBG.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-Qbq3h3VWFkPdtogB.ulx\n",
      "Global seed: 14\n",
      "Global seed: 13\n",
      "Global seed: 15\n",
      "Global seed: 16\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-1QKVfg6YhRb1ul1e.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-jROVIEqEIya6Tr0L.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-7yGrcV9pTE8DF75n.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-6GMVtjVYF5QRupyN.ulx\n",
      "Global seed: 17\n",
      "Global seed: 18\n",
      "Global seed: 19\n",
      "Global seed: 20\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-nrEoHEqgUba7U1nx.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-Mn8oTkr2fvv8TX1.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-VLpEiW2msKJpTZ7J.ulx\n",
      "Game generated: /data/src/TextWorld/notebooks/testing_games/tw-simple-rDense+gDetailed+test-house-GP-o8WVtobyuRR0Uqv5.ulx\n"
     ]
    }
   ],
   "source": [
    "! seq 1 20 | xargs -n1 -P4 tw-make tw-simple --rewards dense --goal detailed --test --output testing_games/ --seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rewardsDense_goalDetailed.ulx..........  \tavg. steps:  95.8; avg. score:  9.2 / 11.\n",
      "./testing_games........................................................................................................................................................................................................  \tavg. steps:  88.4; avg. score:  0.8 / 1.\n",
      "./testing_games........................................................................................................................................................................................................  \tavg. steps: 100.0; avg. score:  0.4 / 1.\n"
     ]
    }
   ],
   "source": [
    "agent.test()\n",
    "play(agent, \"./games/rewardsDense_goalDetailed.ulx\")  # Averaged over 10 playthroughs.\n",
    "play(agent, \"./testing_games/\", nb_episodes=20 * 10)  # Averaged over 10 playthroughs for each test game.\n",
    "play(RandomAgent(), \"./testing_games/\", nb_episodes=20 * 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While not being perfect, the agent manage to score more points on average compared to the random agent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "Here are a few possible directions one can take to improve the agent's performance.\n",
    "- Adding more training games\n",
    "- Changing the agent architecture\n",
    "- Leveraging already trained word embeddings\n",
    "- Playing more games at once (see [`textworld.gym.make_batch`](https://textworld.readthedocs.io/en/latest/textworld.gym.html#textworld.gym.utils.make_batch))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Papers about RL applied to text-based games\n",
    "* [Language Understanding for Text-based games using Deep Reinforcement Learning][narasimhan_et_al_2015]\n",
    "* [Learning How Not to Act in Text-based Games][haroush_et_al_2017]\n",
    "* [Deep Reinforcement Learning with a Natural Language Action Space][he_et_al_2015]\n",
    "* [What can you do with a rock? Affordance extraction via word embeddings][fulda_et_al_2017]\n",
    "* [Text-based adventures of the Golovin AI Agent][kostka_et_al_2017]\n",
    "* [Using reinforcement learning to learn how to play text-based games][zelinka_2018]\n",
    "\n",
    "[narasimhan_et_al_2015]: https://arxiv.org/abs/1506.08941\n",
    "[haroush_et_al_2017]: https://openreview.net/pdf?id=B1-tVX1Pz\n",
    "[he_et_al_2015]: https://arxiv.org/abs/1511.04636\n",
    "[fulda_et_al_2017]: https://arxiv.org/abs/1703.03429\n",
    "[kostka_et_al_2017]: https://arxiv.org/abs/1705.05637\n",
    "[zelinka_2018]: https://arxiv.org/abs/1801.01999"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
